The cellular interactions responsible for dental development involve a complex interplay of signalling molecules during the sequential stages of odontogenesis. Recent studies suggest that bone morphogenic proteins and the related transforming growth factor-? proteins act as signalling molecules during dental development. Two homeobox gene families, muscle segmentation (Msx) and Distal-less (Dlx) genes, also appear to be involved in dental development. However, the complex array of signalling molecules involved in regulating tooth development is far from fully defined. The number of growth factors, receptors, signalling molecules, transcription factors, intracellular molecules, extracellular molecules and plasma-membrane molecules thought to be involved indicates the complex nature of this process.

The presence of serotonin (5-hydroxytryptamine; 5-HT) uptake within dental mesenchyme and dental epithelium has led to the hypothesis that it plays a part in dental development. This possibility is supported by the finding that 5-HT₃ receptor mRNA is expressed in the developing tooth-germ within the epithelium of the bud-stage and within the dental papilla of the bell-stage. Our purpose now was to investigate which of the 5-HT receptors are involved in dental development.

Uptake sites for 5-HT are closely correlated with key morphogenetic events in craniofacial development. Treatment of cultured, mid-gestational mouse embryos with 5-HT uptake inhibitors (sertraline, fluoxetine, amitriptyline) caused craniofacial defects, including mandibular and maxillary hypoplasia, and failure of lens-vesicle invagination. Similar malformations are reported in cultured mouse embryos exposed to 5-HT receptor ligands.

Taken together, these findings suggest that 5-HT may act as a morphoregulatory molecule during craniofacial and cardiac development. This regulation could be by direct effects or through serotonergic regulation of other morphoregulatory molecules. Serotonergic regulation of two such molecules, S-100? (a calcium-binding protein) and tenascin (an extracellular-matrix molecule) was demonstrated in mandibular micromass cultures. Subtype-specific 5-HT receptor antagonists produced differential effects on mRNA and protein expression of S-100? and tenascin. Moreover, two of these antagonists inhibited the expression of cartilage proteoglycan core protein, suggesting that 5-HT might regulate the development of cartilage matrix in vivo. In parallel studies, organotypic mandibular explant cultures were used to investigate the effects of 5-HT on dental morphogenesis. A dose-dependent stimulation of tooth-germ development by 5-HT was demonstrated, which was reversed by the 5-HT uptake inhibitor, fluoxetine.

Here we have focused on the role of three 5-HT receptor subtypes previously linked to serotonergic regulation of gene expression in mandibular micromass cultures, and on 5-HT uptake, which regulates morphogenesis in whole-embryo culture. In both of these culture systems the medium is supplemented with serum, which contains micromolar amounts of 5-HT, making it necessary to use 5-HT antagonists to investigate possible roles of 5-HT. In micromass cultures, the 5-HT_1A receptor antagonist NAN 190 promoted tenascin expression whereas the 5-HT₂ antagonist mianserin stimulated both S-100? and tenascin expression. In contrast, the 5-HT₃ receptor antagonist, ondansetron inhibited core-protein and S-100? expression, but stimulated tenascin expression. The 5-HT uptake inhibitor fluoxetine was used because 5-HT uptake sites are present in the bud-stage tooth-germ, both in the epithelium and condensing mesenchyme. Therefore, we used organotypic mandibular explant cultures (which are grown in serum-free medium) to investigate how dose-dependent effects of 5-HT on tooth-germ and cartilage-matrix development may be related to the serotonergic regulation of S-100? and tenascin. To determine which 5-HT receptors may mediate these effects, subtype-selective 5-HT antagonists and the 5-HT uptake inhibitor fluoxetine were used to reverse the effects of 5-HT.

Cultures were generated from E13 mouse embryos (60–65 somite stage; plug day=E1), as described previously. In brief, timed-pregnant dams were killed by cervical dislocation and the concepti rapidly removed. Embryos were dissected out and their mandibles separated from the remainder of the developing craniofacial region by one cut between the maxillary and mandibular process and a second between the hyoid and mandibular arch on each side, with the two cuts being joined posteriorly. At this stage, the tongue bud is easily visible and was used as a guide to placement of the explant with the oral epithelium uppermost on squares of sterile Millipore filter (pore size 0.8 ?m). These filters were suspended on wire mesh, in contact with a fully defined culture medium (BGJb, no serum supplement; GIBCO, New York), in organ-culture dishes. The culture medium was supplemented with 0.01–100 ?M 5-HT. The medium and appropriate supplements were replaced every 48 h. This range of concentrations would be expected to stimulate a variety of 5-HT receptor subtypes or to elicit 5-HT uptake. The highest dose of 5-HT used (100 ?M) is equivalent to the concentrations found in animal sera. In order to prevent the rapid degradation of 5-HT, 10 ?M -cysteine (an antioxidant) and 10 ?M nialamide (a monoamine oxidase inhibitor) were added with 5-HT.

Explants were cultured in a humidified, 5% CO₂ incubator. Culture medium, with or without additives, was replaced every 48 hr. To assess the ability of 5-HT receptor antagonists or the 5-HT uptake inhibitor fluoxetine to reverse the effects of 5-HT, 1.0 ?M 5-HT was combined with 10 ?M of one of the following agents: ondansetron (Zofran) (5-HT₃ antagonist; Glaxo), mianserin (5-HT_2A–2C antagonist; Research Biochemicals International, RBI), NAN-190 (5-HT_1A antagonist; RBI), or fluoxetine (5-HT uptake inhibitor; gift of Eli Lilly). The dose of 5-HT used (1.0 ?M) was the lowest concentration that had produced recognizable bell-stage tooth-germs in our previous study. Control cultures were either treated with -cysteine and nialamide (10 ?M each) or left untreated. After 8 days in vitro, the cultures were fixed in phosphate-buffered 4% paraformaldehyde, dehydrated in ethanols and toluene, and embedded in Paraplast-plus. Sections were then cut at 10 ?m on a rotary microtome.

Embryos were harvested at two time-points for study of 5-HT receptor expression in vivo. Timed-pregnant dams were killed by excess ether at E14 and E18. The embryos were rapidly removed by a caesarean section, as described above, immediately fixed by immersion in 4% phosphate-buffered paraformaldehyde, embedded in Paraplast-plus, and 10-?M sections cut on a rotary microtome. Sections were deparaffinized and stained by the avidin/biotin peroxidase method (Vector) to characterize the in vivo expression of 5-HT receptors in the tooth-germ and Meckel’s cartilage. Expression of 5-HT receptors was established using antibodies recognizing the 5-HT_1A receptor (1:1000 dilution; Raymond et al., 1993) and an anti-idiotypic antibody that recognizes a number of 5-HT receptor subtypes including 5-HT_1B and 5-HT_2A–2C, but not the 5-HT_1A subtype (1:100 dilution, designated here as non-1A; Tamir et al., 1991).

Antisera to the three molecules of interest were used to characterize their distribution in E14 and E18 embryos: core protein, part of the cartilage proteoglycan extracellular matrix (gift of Dr. Y. Yamada; 1:100 dilution; Doege et al., 1987); tenascin, an extracellular-matrix protein that is associated with the initiation of mesenchymal differentiation and chondrogenesis; and S-100?, a calcium-binding protein that is expressed in glial cells, craniofacial mesenchyme and cartilage.