Demystifying AI

Part of a real sentence is provided, with a word missing. The model is trained to make a guess at what the most likely word is, then shown the real answer, and the model is updated so it’s more likely to guess correctly in the future. This process is repeated billions of times across massive datasets.

1. When a user inputs a prompt, it is first broken down into smaller units called tokens. For the sake of explanation, we’ll assume that each word is a separate token, but in reality tokens can be fragments of words. Each token is pre-assigned a numerical value in the model’s dataset.

2. These tokens are then converted into multidimensional vectors, where the values of each dimension encode meaning and usage into the token. These dimensions are known as parameters, and they can help describe things like the tone of the word, whether it’s big or small, if it’s colorful, etc. Words that are related to each other will have parameters of similar values. GPT-4 has 1.7 trillion parameters.

3. These vectors are processed through layers of transformations using mechanisms known as attention and feed-forward networks. This process allows the model to adjust the values of the parameters for each token based on all of the other tokens. This is what allows the AI to determine the context of how each word is used. In this example, the presence of the token “AI” updates the values of the token “model” from describing a fashion model to describing an AI model. This refinement process is repeated many times, gradually building a more contextually accurate representation of the input. GPT-4 is a 120 layer model, so this process is repeated 120 times. This number is relatively arbitrary - more layers improve the quality of the output, but with the payoff of a higher cost of compute.
4. Now that the data is contextualized, the model presents a probability for what the most likely next token will be, and a token is selected from the list. Up until this point, the process is repeatable with exacting results. Given the same prompt, the model will come to this deterministic state, with the exact same next-token probabilities.

5. Next, a token is selected from the list. The token that is chosen isn’t always the most-likely token. If it was, the AI would produce consistent, and pretty boring results. A tuning mechanism known as temperature determines how likely the model is to deviate from choosing the top result. This is why the same prompt will produce varied responses.
6. The selected token is appended to the text, and the process repeats until either an end of response token is generated, or a compute limit is reached.

Two goals must be accomplished for an image generator to function. The model must learn to render images, and it must internalize the relationship between text and images in a way that allows it to generate a relevant image when prompted.

To begin training, a massive dataset of images with text descriptions is collected - this includes captions, tags, and metadata. For current models, this is essentially the entire content of the internet. (more on this later)

Goal 1 - Predict Images

1. An image is chosen from the dataset, and noise is gradually added to it until it becomes pure noise. This process is called forward diffusion, and it’s repeated a few billion times.
2. The model is then shown a noisy image and a corresponding text prompt. It’s trained to predict the total noise that was added, and attempt to reconstruct the original image.

Goal 2 - Encode Text-Image Relationships

1. Just like the LLM training, images can be broken up into parts and analyzed for patterns that provide context. In this case, tokens are pixels rather than (parts of) words. The color values of the pixels help inform the parameter values of the image, just as the words in the sentence help inform the parameter values of a sentence. A sentence beginning with the words “Once upon a time…” is very likely the beginning of a story. An image where the first four pixels are blue might be likely to depict a blue sky.
2. The dataset is arranged in a massive table, such that the images are on one axis, and the corresponding captions are on another. At this point, the data used for the images is random. Through training, it will be adjusted to usable values.

3. The diagonal of this table contains matching image-caption pairs. To correctly encode the context of the images, the model adjusts the image data so that it maximizes the similarities of the values along the diagonal, while minimizing the similarities elsewhere.

With the combination of these goals accomplished, the model is able to infer what sort of image best matches the prompt being requested. It starts with random noise, and denoises the image in a way that matches the desired prompt.

Interestingly, generating an image is less computationally intensive than generating a text response, as the model does not need to iteratively add on one pixel at a time, in the way that it does with text.

Training large AI models requires significant computational resources, and running them at scale, known as inference, adds ongoing energy demands. The extent of this impact is difficult to measure accurately, as companies do not freely release this data. The best estimates come from concerted efforts to analyze and extrapolate data from smaller open-source models and hardware capabilities. Energy-per-prompt sounds like a straightforward metric for tracking the cost of inference, but it unfortunately breaks down quickly. Unknowable factors like the location of the data center processing the request, the time of day in that area, and number of concurrent requests from other users can cause the energy use to wildly fluctuate. Some prompts even require multiple subsequent prompts within the LLM to generate a response.

While the individual footprint may be small, it is built on top of existing energy demands, and the scale of the implementation is substantial. Estimates vary widely, and the lack of standardized reporting makes it difficult for consumers or policymakers to make informed decisions. As a result, these technologies are being adopted and expanded without a reliable understanding of their true environmental cost.

AI-generated content is contributing to what some describe as a fragile information ecosystem. The rapid production of low-quality or misleading content (slop) blurs the line between authentic and generated material. This not only makes it harder to discern truth, but also creates feedback loops - where AI-generated content is used to train future models, potentially amplifying inaccuracies over time. These dynamics raise concerns about a “dead internet,” where human-generated content becomes increasingly diluted. Humans currently account for only 49% of internet traffic. With the rise of slop, consumers have a growing responsibility to regulate and validate their sources of truth.

In the classroom, AI tools present both opportunities and challenges. While they can support learning, over-reliance may hinder the development of critical thinking and problem-solving skills. An increasing readiness to offload cognitive work, and the ease of generating answers contributes to academic dishonesty, undermining the value of assessments and credentials. Advancement and deployment of the technology is outpacing educators’ ability to adapt their curriculum to it.

Creative professionals face additional pressures, as generative tools can replicate styles and produce content at scale, often trained on copyrighted works without consent, credit, or compensation. Models can be trained to produce art in specific personal styles, and often have the artist’s name attached, again without their consent. This isn't limited to established artists either. As Steven Zapata puts it in his TEDx Talk, “I get messages from [the students] every day… they worry that they’re going to teach an AI system their style before they’ve had a chance to do anything good with it in the first place.” This raises questions not only about the economic impact of but also about the erosion of creative confidence and identity. The optics of using AI generated content in professional settings are already trending negatively, but there is a cost with engaging and supporting such systems even for personal use. There may come a future where AI art is indistinguishable from human art, but it is not a future I’m personally striving for.

In the process of writing this paper, I made a significant algorithmic sacrifice. I consumed an unrecommended amount of information regarding AI technology, and it is now salient in my content feeds. Some of it was fantastic, and a lot of it was not. Here are some sources that I think are worth checking out.