4E091 'Parameters' for an artificial intelligence model trained utilizing 10 26 or more 'operations.'

Technical Notes:

For the purposes of 4E091:

1. 'Parameters' refers to any value learned during training (e.g., network weights, biases, etc.). 'Parameters' for an artificial intelligence model are also known as model weights.

2. 'Operations' refers to mathematical operations used for pre-training and any subsequent training, such as fine-tuning the pre-trained model, but does not include the collection and curation of the input training data. Training 'operations' should account for operations required to perform all steps in the pre-training and subsequent training process, including those for forward and backward propagation for all relevant layers, pooling, and convolutions, regardless of the implementation and hardware limitations, and applied to all relevant operations.

For example, consider a model composed of a single densely connected layer with I input neurons, O output neurons, and no biases being trained with backpropagation.

Such a model would have a total of N = I * O learned parameters.

Each forward pass would require N multiply accumulate operations, or (assuming floating point arithmetic) 2N FLOP. Each backward pass would require 2N multiply accumulate operations, or 4N FLOP. Then, in total, each training data point would require 6N FLOP. Training on a data set of size D would require 6ND total FLOP.

3. If more than ten percent of the 'operations' involve training on data that was not “published” as defined in § 734.7(a) and was generated by a single data-generation model, then 'operations' the data-generation model used to generate the data should be counted, and if the data-generation model's 'parameters' were not “published,” then the training 'operations' used to train the data-generating model should be counted as well. A single data-generation model includes variants of the same model, such as different checkpoints or fine-tuned variants.

If multiple models with 'parameters' that were not “published” were used to generate data used for more than ten percent of the training 'operations,' then only the training 'operations' of the data-generation model using the most training 'operations' should be counted. No actual training 'operation' should be counted more than once, so if data-generation model A is used to train data-generation model B and data-generation model C, and models B and C are used to train model D, then the operations to train A are only added to the number of operations for model D once.