There is a lot of sloppiness in the definitions of literary terms. That's not surprising as the study of literature is not an exact science. However, it is important to know that we're all talking about the same thing. Northrop Frye, in his Anatomy of Criticism (1957), tries to bring some precision to the analysis of fictional forms. In that book's Fourth Essay, "Theory of Genres," he identifies four continuous forms of prose fiction: the novel, the romance, the confession, and the Menippean satire (a/k/a the anatomy).
For Frye, "novel" is not an umbrella term because not all forms of prose fiction are technically novels:
"When we start to think seriously about the novel, not as fiction, but as a form of fiction, we feel that its characteristics, whatever they are, are such as make, say, Defoe, Fielding, Austen, and James central to its tradition, and Borrow, Peacock, Melville, and Emily Bronte somehow peripheral." (304)Characteristics of the Novel: The plot maneuvers around a central situation; "the novelist deals with personality, with characters wearing their personae or social masks. He needs the framework of a stable society, and many of our best novelists have been conventional to the verge of fussiness." (305); "an important theme in the more bourgeois novel should be the parody of the romance and its ideals." (306); "The novel tends..to expand into a fictional approach to history. ... the larger the scheme of a novel becomes, the more obviously its historical nature appears. As it is creative history, however, the novelist usually prefers his material in a plastic, or roughly contemporary state, and feels cramped by a fixed historical pattern. ... most 'historical novels' are romances. Similarly, a novel becomes more romantic in its appeal when the life it reflects has passed away..." (306); "the technical problem [of the novel] is to dissolve all theory into personal relationships." (308); "The novel tends to be extroverted and personal; its chief interest is in human character as it manifests itself in society." (308); "the picaresque form...has the novel's interest in the actual structure of society." (310); "The novelist shows his exuberance either by an exhaustive analysis of human relationships, as in Henry James, or of social phenomena, as in Tolstoy." (311) Examples include: George Meredith's The Egoist, Austen's Pride and Prejudice, James's The Portrait of a Lady, Fielding's The History of Tom Jones, a Foundling, and the works of Defoe and Trollope. The short form of the novel can be found in the short stories of Chekhov or Katherine Mansfield or Joyce's thematically-linked Dubliners.
Romance is not simply the sort of thing one finds under, say, the Harlequin imprint. "The romance, which deals with heroes, is intermediate between the novel, which deals with men, and the myth, which deals with gods." (306) "The romancer does not attempt to create 'real people' so much as stylized figures which expand into psychological archetypes. It is in the romance that we find Jung's libido, anima, and shadow reflected in the hero, heroine, and villain respectively. That is why the romance so often radiates a glow of subjective intensity that the novel lacks, and why a suggestion of allegory is constantly creeping in around its fringes. Certain elements of character are released in the romance which make it naturally a more revolutionary form than the novel. ...The romancer deals with individuality, with characters in vacuo idealized by revery, and, however conservative he may be, something nihilistic and untamable is likely to keep breaking out of his pages." (304-05); "Romance is older than the novel...The social affinities of the romance, with its grave idealizing of heroism and purity, are with the aristocracy... . It revived in the period we call Romantic as part of the Romantic tendency to archaic feudalism and a cult of the hero, or idealized libido." (306). Examples include: Bunyan's The Pilgrim's Progress, Hawthorne's The House of the Seven Gables, Scott's Waverley, Emily Bronte's Wuthering Heights, and the works of William Morris. The short form of the romance is typified in the tales of Poe or Boccaccio's Decameron.
The Confession began with Augustine and found renewal in Rousseau, though its fictional form does not necessarily concern itself with the author: "Most autobiographies are inspired by a creative, and therefore fictional, impulse to select only those events and experiences in the writer's life that go to build up an integrated pattern. This pattern may be something larger than himself with which he has come to identify himself, or simply the coherence of his character and attitudes." (307); "After Rousseau—in fact in Rousseau—the confession flows into the novel, and the mixture produces the fictional autobiography, the Kunstler-roman, and kindred types. ...The 'stream of consciousness' technique permits of a much more concentrated fusion of the two forms, but even here the characteristics peculiar to the confession form show up clearly. Nearly always some theoretical and intellectual interest in religion, politics, or art plays a leading role in the confession. It is his success in integrating his mind on such subjects that makes the author of a confession feel that his life is worth writing about." (307-08); Like the romance, "[t]he confession is also introverted, but intellectualized in content." (308). A notable example: Joyce's A Portrait of the Author as a Young Man. The short form of the confession is what Frye calls the "familiar essay" perhaps best exemplified in the Essays of Montaigne.
Finally, the Menippean satire, or what Frye calls the 'Anatomy', "deals less with people as such than with mental attitudes. Pedants, bigots, cranks, parvenus, virtuosi, enthusiasts, rapacious and incompetent professional men of all kinds, are handled in terms of their occupational approach to life as distinct from their social behavior. The Menippean satire thus resembles the confession in its ability to handle abstract ideas and theories, and differs from the novel in its characterization, which is stylized rather than naturalistic, and presents people as mouthpieces of the ideas they represent. ... A constant theme in the tradition is the ridicule of the philosophus gloriosus ... . The novelist sees evil and folly as social diseases, but the Menippean satirist sees them as diseases of the intellect, as a kind of maddened pedantry which the philosophus gloriosus at once symbolizes and defines." (309) "At its most concentrated the Menippean satire presents us with a vision of the world in terms of a single intellectual pattern. The intellectual structure built up from the story makes for violent dislocations in the customary logic of narrative, though the appearance of carelessness that results reflects only the carelessness of the reader or his tendency to judge by a novel-centered conception of fiction." (310); "The Menippean satirist, dealing with intellectual themes and attitudes, shows his exuberance in intellectual ways, by piling up an enormous mass of erudition about his theme or in overwhelming his pedantic targets with an avalanche of their own jargon." (311). Examples are rife in Frye: the works of Petronius and Rabelais, Burton's The Anatomy of Melancholy, Walton's The Compleat Angler, Swift's Gulliver's Travels, Sterne's Tristram Shandy, Voltaire's Candide, Flaubert's Bouvard et Pecuchet, Huxley's Brave New World, "The short form of the Menippean satire is usually a dialogue or colloquy, in which the dramatic interest is in a conflict of ideas rather than of character. This is the favorite form of Erasmus, and is common in Voltaire." (310). The short form derives from Plato's dialogues, is found in Boethius's The Consolation of Philosophy, and often involves a symposium or a country-house weekend.
The forms are rarely pure; many works of fiction are mixed. "[T]he early novels of George Eliot, for instance, are influenced by the romance, and the later ones by the anatomy." (312) Romance and novel converge to ironic effect in such works as Northanger Abbey, Madame Bovary, and Lord Jim. Moby Dick is a romance-anatomy. de Quincey's Confessions of an English Opium-Eater is a romance-confession hybrid. Confession and anatomy are united in Sartor Resartus and, surprisingly, in Kierkegaard's Either/Or (sorry, it's the best I could find). Novel, romance, and confession come together in Pamela; novel, romance, and anatomy in Don Quixote; novel, confession, and anatomy in Proust; and romance, confession, and anatomy in Apuleius. Joyce's Ulysses Frye calls "a complete prose epic with all four forms employed in it, all of practically equal importance, and all essential to one another, so that the book is a unity and not an aggregate." (314).