I do not consider myself an OSR person, though I do play OSR games on occasion, and I appreciate the movement's DIY spirit, openness, design edge, and philosophy stipulating that "nothing is supposed to happen". I lack the requisite reverence for the hobby's Founding Fathers, and am at best lukewarm toward OSR's dominant dungeon-and-frontier aesthetic. I also think randomness is a necessary element of any RPG, and that it is of equal importance to a GM's worldbuilding and narrative skill, and player agency; but I do not believe randomness should be an imperative in itself - only a tool, and a mediating element between GMs and players. I don't think everything about a character needs to be randomly generated, and while character death and maiming need to be part of the game, the most interesting characters are those that are built with care, not those that are designed as cannon fodder and happen to survive.
In general, I think that DIY is applicable to all RPGs, and certainly, has always been a mainstay of D&D in all its incarnations. Not all systems are appropriate to all settings, but for most fantasy save truly low-magic settings, most types of D&D and its offshoots are generally suitable. On the whole, I think 5e is a fine system, that deserves the accolades it has received. It has simple, streamlined, easy-to-learn mechanics. It has a sufficient number of basic player options (classes and races) to make it attractive to a large and expanding player base, and the fact that the player options are designed from the flavor up, and not from the mechanics up, is a definite point in its favor. The design system is also fairly easy to reverse engineer, and serves as a good guide for DIY race and class design. Moreover, the introduction of backgrounds has proven to be a very useful feature that helps to ground characters in settings. And the elimination of earlier mistakes - the lack of integrated mechanics of AD&D, the crunch-heaviness of 3e, the 4e attempt to become a low-tech tactical video game - has given 5 justified recognition as a back-to-basics game that even OSR people have appreciated. And ultimately, 5e's popularity which some gamers complain about, has created an unprecedentedly large pool of potential players, some of whom can be poached for less conventional and more experimental approaches.
At the same time, there are aspects of 5e that I like quite a bit less - not so much the rules, mechanics, and flavor themselves, but the culture they are informed by, and in turn, generate. First and foremost, this involves the obsession with balance. Ideally, design should attempt to mechanically balance distinct classes and races against one another - all should have things they excel in, all should have player appeal, none should be far-and-away better than the others (else, why play them?). Such balance is fine as a goal, but a lot of people fail to realize that the attainment of balance in a complex game like D&D is impossible, simply because it is impossible to quantify every in-game situation. It is widely recognized that some classes are stronger than others, and that, in the grand scheme of things, casters are more powerful than non-casters at high levels (and that combat classes, conversely, are more powerful at low levels). This has recurred in every edition, despite the explicit push to minimize the imbalances starting with 3rd edition.
The real problem with balance is when it begins to bleed over from character design into every other aspect of the game. All encounters must be balanced. The number of encounters per day must be balanced. Each session should be balanced between role-playing, exploration, and combat. Agency must be balanced between players and GM, and all individual players, so as to maximize player efficaciousness over the course of a session (though it's really about efficaciousness more than balance - the same people who complain about a Charm spell being ineffective because NPCs remember you cast it on them will complain that a Charm spell cast on them takes away their PC's agency). The advent of the ideally balanced game session - where the party is challenged, but only just; no one dies; and everyone has a prescribed level of 'fun' is reflected in numerous advice videos, which create certain expectations of what is supposed to happen in a game session or campaign (a 'full campaign' is one in which a character advances from 1st to 20th level, where the 'capstone' is reached). It is therefore not accidental that the ideal game begins to look more-or-less the same regardless of where you look. Heroes, who almost always survive, are railroaded through very similar settings, on route to a fight with the aptly named Big Bad Evil Guy (BBEG), and then retire (occasionally, are killed in a final blaze of glory). The party mix is always the same diverse mix of races and classes that don't really belong together, and don't really belong in the setting, which reduces more and more to generic fantasy. This ideal type of D&D game is somewhat reminiscent of an ideal game of Diplomacy from the point of view of the realist international relations theorist. Ideally, in Diplomacy, skilled players are so sensitive to a balance of power that the slightest imbalance in favor of one Great Power will lead to immediate bandwagoning, so that boundaries don't change. Of course, real games of Diplomacy hardly ever work out that way (as do real interstate relations). Similarly, real D&D, in whatever iteration, is going to be unbalanced. To my mind, this should be factored in, and recognition of the fact can make a game more fun than an adherence to the notion of optimal balance.
Below are some ideas for playing 5e far from equilibrium, which, if done right, can create more unique and memorable campaigns. (I say campaigns, because balance is much easier to achieve in individual sessions, which consist of one set of players, where parties have clear goals and are usually rushed through preliminaries to get to them. In campaigns, where the group chemistry is always in flux, and where the setting changes, and becomes more real over time, the pursuit of balance becomes significantly more challenging. But campaigns are a unique feature of RPGs [whereas Diplomacy always reverts back to 1901 at the start of each discrete session]. Campaigns, in other words, offer quite a bit more scope for DIY).
- Roll for stats and HP. Not all characters have to be the same. Rolling for stats creates tension and can limit choices, but the limitations likely results in more memorable, suboptimal characters (as well as more memorable superoptimal, and even more memorable average ones). A character's choices are impacted by rolls, and the player can choose to adjust accordingly, e.g. by taking a 'Tough' feat as a result of consistently bad HP rolls, or picking a particular archetype because the wizard ends up with a high Strength. Note that having some choice over character type is important - taking the time to make a backstory for a character kind of necessitates not being forced into a particular class and race as a result of a die roll.
- System. Different systems for generating stats are widely known, and need not be detailed here. 4d6, drop the lowest, and arrange in any order works well enough for me, but 3d6 down the line has a certain charm that works for some games. I'm personally not in favor of setting a floor, below which a player is entitled to a re-roll - I'd rather make a deal whereby a really poor character can qualify for later adjustments due to being young, or perhaps give that character a magic item. But if rolls can simply be cancelled, there is no reason to do them. Of course, you have to make sure that players understand this ahead of time.
- Ditch the healing surges. 4e introduced them, trying to make D&D more like a video game where you can observe the "health bar" going from red to green. These healing surges are what allow campaigns to transpire in an entirely abstract atmosphere - no wounds are more than bumps and bruises, no healers are necessary (any character can just heal themselves, because balance), and no relationships with NPCs that might help a character's predicament need be established. Everyone is self-sufficient. The surges are also central to the assumptions upon which classes are balanced - getting rid of them introduces the (rather realistic) factor that characters trained specifically for combat are better at shaking off battle effects (the fighter's second wind feature). Healing surges also make the game more combat-oriented. The recommended 6-8 combat encounters per day (that hardly any campaign-style game actually implements) are also based on the availability of healing surges, which are a key baseline in resource management. If a single fight can stretch resources, managing them becomes more unpredictable, and forces parties to change tactics to incorporate more negotiation, cunning, and speed, and less murder.
- System. Simply use the gritty healing variant already available in the Dungeon Master's Guide: a long rest (8 hours) makes hit dice available to restore hit points, and a period of rest (or downtime) lasting a week or so makes it possible to recover hit dice. This preserves a modicum of the surges, and allows characters to be self-reliant in a pinch, but it greatly restricts free-floating cures. For more granularity that helps ground characters in a setting, make the hit dice recovery subject to the quality of the rest - if it's "poor" or "squalid" (as per the categories listed in the Equipment chapter), the character can recover 1 or 0 hit dice per week, and if it's "comfortable" or "wealth", allow 3, 4 or more hit dice per week. Suddenly, figuring out how to stay in a mansion is no longer a meaningless luxury.
- Relatedly, use Lingering Injuries. The argument that hit points represent fatigue, skill, and luck has a long pedigree, going all the way back to Gygax. But hit points were also physical damage. Even in 4e, a character that lost half of its total hit points was considered "bloodied". Characters that are knocked to 0 hit points are not just 'tired', but have likely sustained some sort of wound. The argument that damage is all virtual falls down on several counts. First, there is now a mechanic for Exhaustion, which is a less granular (though to my mind, effective) way to track hunger, impact of the elements, and actual fatigue. Second, if a damage type (e.g. "slashing" or "bludgeoning") is listed (as it has been in all rulesets since 3e), the idea that all damage is virtual doesn't stand up (else, why distinguish it?). As with controlling healing surges, making damage dangerous obviates a lot of common assumptions about balance, and pushes parties to be more creative about tactics.
- System. The table provided in the DMG is rather stripped-down, and rarely leads to serious complications, but it is one option. Creating the possibility of lingering injuries on a critical hit, or when a character is reduced to 0 hit points seems reasonable. The severity of the injury can be linked to the amount of damage suffered, number of Death Saves failed, whether the hit was a critical and knocked a character out, etc. System Shock - another variant rule that recalls an old-school mechanic - is a bit too brutal (even for me) at low levels (characters make a Con save if they lose half of their hit points or more on a single attack). But if linked to critical strikes, using System Shock can make sense. Making criticals more dangerous (and flavorful) than simply doubling damage also adds extra ooomph to Champion Fighters, which are commonly considered a bit weak and boring.
- Use reasonable assumptions for missile weapons. Lots of people think they're entitled to fire projectiles into melee without a chance of hitting your friends (because the game is balanced upon the assumption that you can). However, using the 'rulings, not rules' guideline that this and earlier editions subscribed to, you can impose a reasonable penalty for doing so. This has the effect of forcing players to actually think about tactics, instead of just pumping out DPR. One side effect this has is to dampen the use of at-will cantrips without completely breaking 5e game design and taking these away from casters. If you are worried about hurting your friends by spamming fire bolts, you either have to take a chance on rotating into combat with a melee weapon or touch spell, or rely on other abilities. Imposing penalties for firing into combat will also impact players' choices of cantrips.
- System. The easiest thing to do is to impose disadvantage on a missile attack into melee, where friends are likely screening an opponent. If either roll is a "1", you roll an attack against your ally. You can make the system more complicated by determining how much cover an ally provides to an opponent, and make the likelihood of hitting the friend higher if your ally provides 3/4 cover (e.g., if either roll is a natural 4 or less).
- Impose limits on the use of magic. Taking away at-will cantrips seems unwarranted, because they are so integral to the system. But other traditional ways of limiting access to spells may work and may be justified, especially in light of the fact that melee specialists are putting themselves in the way of more lingering injuries.
- System. For wizards, you can reintroduce the old mechanic of understanding a spell before being able to copy it into your spell book. Understanding can have a DC, which is modified by proficiency bonus, ability modifier, skill in arcana, and the level of the spell. The wizard must also use a Write spell to copy a newly understood spell into a spellbook, and make a Dexterity check in the process (to make sure no smudges are created). People who complain about having nothing to spend money on in 5e will soon discover their stationery costs beginning to pile up. For warlocks, on the other hand, the recovery of spell slots after a short rest can be linked to a coven feature. A coven has a collective mind, and you tap into it to have access to the extra slot after a short rest. The bigger the coven, the greater the chance that a slot is available. The drawback is that sometimes, coven mates make the same requests from you, and if you refuse them, you will put yourself in a bad position to request spells from the coven mind. This not only encourages resource management, but also creates new role-playing opportunities (coven recruitment is obviously in your warlock's self-interest). It might be a good idea to create more openness even as you impose more limitations, e.g. by allowing a caster who has used up all spell slots to cast a spell by making a DC roll (modified by spell level, and number of the attempt at doing so) at the risk of being subject to damage or other effects if you fail the roll.
- Take terrain and climate seriously. Balancing mechanics typically presuppose an abstract space in which adventuring takes place (which in turn further inclines GMs to set adventures in dungeons). But forcing PCs to negotiate wilderness actually means using a lot of rules that are already on the books, but that GMs hand-wave as they push the party along toward the 'adventuring site'. Make the foraging rolls mean something. Make the navigation rolls mean something. Throw in some terrain hazards (ravines, rockslides, quicksand, wide rivers, etc.) into your encounter tables. Make the characters roll (or undertake actions) to find the hidden tower, instead of just assuming they do. Once characters begin to go without food, acquire exhaustion levels due to being underdressed, or fail to rendezvous NPCs on time because they got lost, parties will begin to make better plans for travel. Also, all of a sudden, rangers' Natural Explorer feature will no longer seem like a bunch of useless fluff, but a highly valued and desired set of skills. If realistic biomes are used, rangers will also know which type of preferred terrain to take at low levels (e.g. because they live in the taiga, instead of bumping into an Old Forest from time to time, when the GM is so inclined). The feeling of living in a certain terrain changes the experience of a game quite drastically from one where it is always a dry 70 degrees Fahrenheit, with perfect visibility.
- System. Again, use the Survival checks for finding food and navigating in specific terrain types that the PHB and the DMG already include. Put hazards into your encounter tables. Keep track of exhaustion due to food and water deprivation, or heat and cold, and make wind impact projectile use, as it should.
- Design character options for a unique setting. So far, we've mostly been talking about limitations, but a far-from -equilibrium game is fundamentally about variance. That doesn't mean reinventing the wheel in every campaign, but it does mean developing a theme. Look at the class and race options in the official sources. Decide what resonates with your campaign, and what doesn't. Keep what works, and design new options based on your favorite historical setting, mythology, fantasy series, film, etc. The new options will have to be playtested, and not every variant will work, but the testing itself pushes the game away from equilibrium, and makes people's characters more experimental. A different range of options also alters party balance. Most importantly, different character options make settings unique. One common complain about each edition is that players demand variants, and new variants create bloat, that in the longer run can only lead to designing new editions. It also makes settings incoherent - more options is automatically assumed to be 'better'. I don't necessarily agree with those who think it's all about making parties more human, and less firbolg or yuan-ti. It's more about making firbolg or yuan-ti thematic, having different sets of options (which include limitations as well as augmentations), not just more of them. As different variants and settings proliferate, they acquire a life of their own, and such differentiation can go on indefinitely. There is no reason why 5e can't have the same number of "hacks" (or more) as OD&D (the existence of the OGL is a wise partial retreat from all-embracing corporate control).
- System. Use existing designs as guidelines and models. There is no reason to make character options unbalanced on purpose - just see what sorts of properties emerge in various games and settings by trying new things, or tinkering with existing chassis. If need be, use a good point-system, or more qualitative approaches to good class or race design. Reintroduce older notions of classes being actual structures within gameworlds, rather than just bags of mechanics. Make access to higher levels limited by number of spots, approval of elders, or results of single combat. Figure out features for "epic"-level characters beyond 20th level.
- Reward different playstyles. If you want different playstyles, you have to valorize them. It's not just about killing monsters, or routing the Big Bad. It can be about negotiating wilderness hazards, finding treasure, stealing important items, negotiating, finding ancient ruins or portals, learning secrets about NPCs, saving lives, and so on. When winning combats is not the default assumption, the game begins to balance very differently. Characters who specialize more in exploration or RP suddenly become more viable. But more than that, the sweet-spot upon which the balance rests constantly changes, depending on the decisions players and characters make. And consequences of particular actions shift the balance in certain directions over the long term. One incautious slaughter of NPCs can lead to more frequent fights, while acquiring a reputation of being a fair dealer, or having a silver tongue, will likely create more opportunities to persuade opponents.
- System. Those who prefer the 'milestone' approach can reward player actions as they see fit. Those who prefer a more traditional XP-based approach can institute a more granular system of rewards. For my part, I usually award 10% of monster XP for mere interaction, 20% for fooling or persuading encountered creatures, and 50% for defeating them without physically vanquishing them (e.g. causing them to flee or retreat). I also give half the monetary value of treasure as XP, but without differentiating whether that treasure was looted, stolen, or paid as stipulated by a contract. Learning information yields a given amount of XP, depending on whether the information was common, uncommon, rare, etc. And making advances within an organization or a social unit is usually worth 1 point of XP per person who belongs to the organization, multiplied by degree of advance (or change of the existing institution). Obviously, there is a whole host of permutations, but you can opt for a system that rewards PCs for exploration and RP without forcing them into combat, which allows for somewhat slow level advancement at low levels (keeping a campaign low-magic, and gritty for longer periods of time, if that's what's desired).
Obviously, the list can be extended indefinitely. Not all recommendations will find purchase with everyone, but the creation of niches can be a way forward at times when a system is threatened by a saturation of interest, and bloat.
