job 18002677 starting on gpub089.delta.ncsa.illinois.edu ==PROF== Connected to process 810795 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_11) ==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 2: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 4: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 5: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 6: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 7: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 8: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 9: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 10: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 11: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 12: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 13: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 14: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 15: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 16: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 17: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 18: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 19: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 20: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 21: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 22: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 23: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 24: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 25: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 26: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 27: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 28: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 29: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 30: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 31: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 32: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 33: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 34: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 35: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 36: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 37: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 38: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 39: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 40: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 41: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 42: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 43: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 44: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 45: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 46: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 47: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 48: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 49: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 50: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 51: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 52: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 53: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 54: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 55: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 56: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 57: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 58: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 59: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 60: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 61: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 62: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 63: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 64: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 65: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 66: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 67: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 68: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 69: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 70: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 71: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 72: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 73: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 74: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 75: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 76: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 78: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 81: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 83: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 86: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 88: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 89: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 90: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 91: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 92: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 93: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 94: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 95: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 96: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 97: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 98: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 99: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 100: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 101: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 102: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 103: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 104: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 105: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 106: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 107: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 108: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 109: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 110: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 111: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 112: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 113: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 114: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 115: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 116: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 117: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 118: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 119: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 120: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 121: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 122: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 123: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 124: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 125: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 126: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 127: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 128: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 129: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 130: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 131: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 132: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 133: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 134: 0%....50%....100% - 40 passes ==PROF== Profiling "permute_kernel" - 135: 0%....50%....100% - 40 passes ==PROF== Profiling "flash_attention_forward_kernel" - 136: 0%....50%....100% - 40 passes ==PROF== Profiling "unpermute_kernel" - 137: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 138: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 139: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 140: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 141: 0%....50%....100% - 40 passes ==PROF== Profiling "gelu_forward_kernel" - 142: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 143: 0%....50%....100% - 40 passes ==PROF== Profiling "residual_forward_kernel" - 144: 0%....50%....100% - 40 passes ==PROF== Profiling "layernorm_forward_kernel" - 145: 0%....50%....100% - 40 passes ==PROF== Profiling "matmul_forward_kernel" - 146: 0% ==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. ....50%....100% - 40 passes test-gpt2-kernels - TF32 Mode is turned OFF. test-gpt2 - [State] Batch Size: 4 Sequence Length: 64 GPT2 - Allocated 221 MiB for activations test-gpt2 - Printing 10 Expected vs Actual Logits: -43.431618, -43.431740 -39.836346, -39.836452 -43.065910, -43.066021 -42.828045, -42.828144 -43.529541, -43.529644 -44.318398, -44.318497 -41.227425, -41.227528 -41.270760, -41.270863 -42.541393, -42.541523 -42.394997, -42.395119 ---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- _________________________ All tests passed! _________________________ ==PROF== Disconnected from process 810795 ==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_11.ncu-rep