Theories of historical change
In the 1990's from the perspective of a kid in the US, the world seemed like it was on the verge of solving all major problems. Neoliberalism seemed to have delivered most of what it promised, and there were just a few pockets left where its positive effects hadn't reached. This seemed to be the attitude of the west in general. Then things took a sharp turn in the early 2000s and in the last 20 years, there has been a steady acceleration of problems popping up, and these days it seems we've been losing our game of whack-a-mole to stop these events from spiraling out of control. Nowadays suddenly people seem pretty pessimistic about avoiding WW3, whether the US will exist in a few years, whether our basic freedoms will be preserved, whether we can turn away from ecological collapse, etc. People on all sides of the political spectrum agree on many of the problems, and feel a similar sense of urgency, but for some reason we are not able to come together and face our problems in an organized way. Why not?
Depends on what you think the major drivers of history are. There are a few major theories that are competing right now and they are the result of different outlooks on life. They result in different opinions about how to stop our momentum toward self-destruction. From what I can tell, they are as follows:
First there is great man theory. This is the idea that the world's fate has been determined by great leaders who push their way to the leadership positions of large institutions and then drive them to change the world. And that when they're good, they improve people's quality of life. This theory was prominent in the 1800s, and is also prominent today in the right wing. It goes hand in hand with social darwinism, because a society with a good genetic pool and good social values will produce more "great men." The society with more great men will be more innovative, gain a competitive advantage, and then overtake the societies without great men who flounder.
This is one of the theories taught in our social studies classes. We learn that George Washington was a great man... he could not tell a lie about chopping down the cherry tree, and that resulted in him being an effective leader, which in turn resulted in him along with other great men--some of whose names we were taught--defeating the forces of the non-great George III. He owned slaves, which is bad, but we're assured that he was probably not too bad to his slaves, and people didn't know it was bad back then, so he was still a "great man".
It emphasizes individual contributions, while de-emphasizing the context--the reason why those individuals were in a position to give those contributions. It also de-emphasizes the bonds between individuals. The logical conclusion is that non-great men are just in the way of the great men. They're sucking up resources and should get out of their way. In order to prop up this idea, it's critical to give examples of the adversity that the great men have come out victorious against. It's also imperative that the great men do not look weak or incapable. Much worse than any moral failing is a blatant loss that was never redressed.
I'm sure there's some value to this idea. It's probably true on a small scale for some kinds of change, but there are some people just waiting around for a great leader to come solve our problems, and I think we can be pretty confident that that's not going to get us out of our current set of problems. Anyone who is good enough to want to seriously change things would never get to power, and anyone with enough power just won't have a realistic view of what problems people are having. To me, the most concise criticism of this theory comes from Mal in Firefly: "It's my estimation that every man ever got a statue made of him was one kind of sommbitch or another".
Then there's what I would call the public consciousness theory or "great idea" theory. This is the idea that society has been slowly increasing in public consciousness and to effect change to improve people's lives, we need more education and "development." This lives symbiotically with the great man theory. Usually great ideas come from great men. A society with good ideas will have better values, and create better men. The reason I find it distinct enough to put in its own category is that it has a shift of focus from the individual.
This theory is also taught in our social studies classes and is the basis of neoliberal thought. The great man who came up with the cotton gin was Eli Whitney, and his great idea was the cotton gin. Similarly, women's suffrage was a great idea. The "great man" who thought of it was Susan B. Anthony. Science is among the greatest ideas, and it is a collaborative process. Of course in science, there are many great men, but their individual greatness is de-emphasized and their ideas take the foreground for neoliberals.
Where the conservatives believe that there are great men and not great men, and those who are not great can only be turned great by trials of extreme adversity, the neoliberals believe that everyone has the potential to be at least a good man, as long as they espouse great ideas. If they never had the chance to be exposed to great ideas, then they can't be blamed for their mistakes. Thus it relies on the idea of historical ignorance to maintain the claim of historical moral righteousness.
In order to prop up this idea in the face of the clear knowledge that many horrible ideas were held by the society it's being applied to, people who hold this view like to emphasize how bad people had it before, and how far we've come. The fact that we work far more hours than pre-industrial society is a surprise to many people who hold this view, and they'll quickly counter with some infant mortality rate statistics. The idea that some people might like to trade a little length of life for quality of life is subversive because it implies that we don't have the highest imaginable quality of life. These people are also infuriated by questioning the institutions which are the designated idea holders. You can trash individual scientists, but you will not trash science. Meanwhile the great men people will support science one sentence and abandon it the next if that's what their favorite scientist is doing.
Of course there's some truth to this as well... our lives are less physically demanding than they were before. Education does seem to improve people's lives. Methodical practices such as science and engineering are great tools. However, the problem comes when people don't think very hard and believe silly ideas like slavery was abolished because people finally realized that it was bad... as if there haven't been people screaming that at the top of their lungs since the first slave was sold. It's missing some understanding that there has always been a vocal opposition that gets ignored. The ideas have all always been in plain sight. There's just a reason power structures haven't adopted those ideas.
Then there's the material theory, which fills that gap in a little bit. The far left espouses this idea that history has all been a struggle between those who have power and resources and those who do not. The ruling class suppresses the lower classes by essentially paying off or manipulating other classes.
In this theory, almost everything in history was the result of the distribution of resources among the people and the power interests of the time. Despite the fact that this view is poo-poohed by the conservatives and liberals, it's clear that people see this view as more gritty and realistic. All the good movies which show significant (in-universe) historical change depict resistance from marginalized groups these days.
The conservatives--especially the far right ones--seem to have a strange respect for this view despite vehemently opposing it. Maybe because the great man view tacitly acknowledges that there is an underclass at least, just one that needs to be either eliminated or dominated and used for the purposes of the great men.
This theory relies on the nebulous idea of organizing and states that the class which is organized will be able to use their capital to gain more power. This puts an emphasis on the bonds between individuals in a way that the great men and great idea theories do not. It also emphasizes context in the actions of an individual. To someone espousing this view, a great man like Napoleon might have been an inevitable consequence of larger social forces. They might think he was just an avatar for a group of people who he represented.
A problem with this set of ideas is that they don't acknowledge much in the way of external forces that can affect society. They also provide no mechanism for how the resolution of class conflict will result in a steady-state increase in quality of life. Furthermore, it fails to explain the lack of historical precedent for class equality, except that capitalists ruin it. But this seems to indicate that it is a very fragile equilibrium, and history is a long time to prop up a fragile equilibrium.
Other ones might be the great genes theory, which ultimately is a form of great men theory. A cyclic view of history or a chaotic view of history (as in the butterfly effect), both of which put no power of change in the hands of people. A moral crusade view of history--where everything comes down to forces of good and evil--ultimately another form of the good ideas theory. Then there are the religious views of history--mostly also a form of the good ideas theory, but with a slightly different twist. These can get much more interesting than the other views listed because they come prepackaged with a set of those good ideas which delve into things like family structure, economic behavior, good leadership, etc. Their prescriptions for these things can point to something deeper than the other theories listed, and at least compared to the standard good ideas theories, they avoid falling into the trap of calling every current idea good because of some inherent belief that we're progressing toward better ideas.
In the view of Judaism, god destroys nations which do not follow those great ideas. Buddhism says that miserable cycles will continue until people believe the great ideas. Confucianism and Taoism outlines the great ideas and basically just state that society will decay if these ideas are not followed.
But the great ideas that the religions mention provide a set of instructions on how to transact with strangers and maintain a healthy society. While they seem obvious on some level "thou shalt not kill" in a case where you're avenging the murder of your brother, it's actually not clear at all that that's the right answer without it being written down. But it's also necessary to stop family feuds from spiraling out of control and destroying people's ability to survive. Similarly, you'd think the message to take care of the poor in society would be clear based on the emphasis in so many religions, and the negative consequences of ignoring it seem obvious, but apparently it's still lost on those in power. The "great ideas" in these cases outline the social bonds that are needed to reproduce and maintain a society, and the conditions for creating those bonds.
These instructions implicitly point to a mechanism for at least some historical changes. By not following the rules that maintain social bonds and social order, the bonds break down and so the social organization collapses. This is a more concrete mechanism than any of the others given and so has more explanatory power. Even the material view seems to be very vague about how the changes in class conflict play out, other than just labeling the actors involved and their class interests. Meanwhile "great man" and "great ideas" theories end up falling apart when you try to define what makes them "great."
There are more academic views as well, from sociology, etc. They generally involve analyzing contextual factors. For instance many major collapses are blamed on natural disasters, foreign invaders, plague, or climate changes. Often a handful of these are put together and get the blame. These are essentially equivalent to chaos theory in that they basically say there's no way to make any predictions about what will happen... something that's obviously not true because life and consciousness depend on anticipating events in the future and constraining those events through behavior. A more generalizeable analysis called "the collapse of complex civilization" (by Joseph A. Tainter) I've read goes through the weaknesses of these arguments in the introduction, and then provides its own compelling arguments for why civilization collapses, partially by turning the question on its head: why does civilization ever pop up to begin with?
Its main argument (combined with some of my editorializing) is that in some cases the complexity becomes worth it because it yields returns. At first those marginal yields of more complexity increase as economies of scale take force, but eventually they start decreasing as the low-hanging fruit is consumed, and then eventually the gross returns decrease and the complexity costs more than it yields. For instance, a complex organization might pop up to distribute water from a central source to surrounding areas that are comparatively dry, while those surrounding regions provide building materials and food to the water-heavy area. The fact that complexity in this case is cropping up to reduce the steepness of a gradient matches very well with the idea of life and society as a dissipative structure. That gradient is a potential difference, and thus can be used to do work.
If we imagine the complexity that pops up is a group that builds cisterns and carries water regularly to those cisterns and brings back building materials from these areas, then we can see that in the areas that were drier before, there is now more water. This is a form of diffusion. Maybe the people working at one cistern get better building materials than another. They come back to the village around the water source and make a better house, they have more status. They tell people "let's build more cisterns out that way," because it worked for them, and of course people listen because they want a nice house like they got. So people build more cisterns out there and the surrounding areas suddenly can support more life, so create more demand for cisterns. The ones who made that decision are gaining more influence over the decision-making process as long as this strategy works.
However the gradient is also less steep because there is water flowing out that way via previous cisterns, so as they build more cisterns, it gives fewer returns for the amount of work. More teams are required. Now they're carrying water further out and maintaining more cisterns. But the area with the water is now supporting more life as well, meaning they have to drink more of their own water. Eventually they get to the point where they're not getting much from building more cisterns and the water in their home is starting to dry up, but who has influence over the decision-making process and what are their interests? Obviously the waterboys are in charge and, now they're getting a little desperate, so they're more likely to cling to the their power. And they're a one-trick pony. It's easy to see how this little society would be locked into their pattern. Their solution to all the problems bubbling up is "build more cisterns".
So that's a description of one kind of historical change: namely collapse, but its methodology and logic is applicable to other kinds of change. Its methodology is analyzing the marginal return on complexity. This "complexity" I think is basically the same thing that a materialist view would call "organization." There is no explicit description of intentional change. Essentially it states that organization/complexity happens where it is possible, and the default state is that not being possible.
So we need to synthesize a view of history that allows for intentional change while drawing thorough causal chains for how actions will result in change. Just being a great man is cool, but most people have their hands tied paying bills and can't really do that. Having great ideas is nice, but usually they're just ignored by anyone with enough influence to do anything about them. Organizing sounds easy enough, but organize into what structure? Color coded? Alphabetical?
What is missing from the collapse scenario with the water people? Information from the ground directly influencing decision-making. The original water boys solidify their influence, and any new information coming into the system does not come from a source with enough influence to re-direct the strategy. Democracy tried to solve this by having votes... This is essentially redistribution of influence among people who may have better ideas of what's going on.
However, how many bits of information does each person give in direct influence over the decision-making process? Very few. So influence is still concentrated into a few hands who push other people from influence. Our society is fractal, so this pattern repeats itself on many levels. This is true for businesses, where the founders get a large amount of equity, while giving some of it up to rich people. This equity represents both control of resources as well as influence over the direction of the company. However for a large company, these people with the equity might have very little understanding of what's happening on the ground, and they have very little short-term incentive to care.
Since society is fractal, that means if change will happen, it will happen on many levels simultaneously. It also means that a massive change on one level will propagate to the other levels. Basically one way to truly effect a change in our society would be to encourage more formalized bottom-up information inputs into whatever institutions we're actively participating in. Whatever decision-making processes they have, if those processes can be encouraged to actively collect and digest decision-making input from less-influential members, not just collecting their data surreptitiously and making decisions for them, this could result in more ability for us to move ourselves out of harms way as a society.