5 forecasts: the price of oil in December as our target forecast, conditioned separately on the status of the Strait of Hormuz and the result of the US Senate election.
I love Mantic's approach to forecasting, but I'm confused about how these Friday Forecast blogs are structured. Are these the direct chain of thoughts of the LLM or simply the final output that reflects on the chain of thought?
Relatedly, how are the quantitative constraints structured? Noting that "I accounted for oil’s inherent volatility by maintaining wide 'tails' in my distribution" still leaves ambiguity as to the extent of the tails as well as the presence and direction of skew.
Mantic's forecast output consists of a probabilistic forecast that depends on the question being asked (e.g. a categorical distribution over candidates in an election or, in this case, a probability density over the price of oil) and a summary of its reasoning in prose.
The bullet points in the post are extracted directly from the summary prose. The text in italics is added as background and is not produced by the system. Quantitative descriptions are made concrete in the probability distributions that are produced.
I understand if this is a proprietary technique that can’t be divulged, but the method for transforming CoT reasoning / prose into quantitative parameterizations for a log-normal distribution is unclear to me.
In Scott’s talk at ICML, he mentions that the harness prompts the model to output a mixture of log-normal distribution parameters. However, that seems to grant the model enormous numerical freedom, both in choosing the log-normal parameters and the weight given to different scenarios.
For instance, how does the model discern that $mu$ of Scenario A should be $x$ instead of $x + epislon$ or $x + 2 * epsilon$ ? Or, how does the model determine that Scenario B should have weight $w_b$ instead of $w_b + epsilon$ ? (where epsilon is some arbitrary real number)
There are some proprietary techniques at work, but here's a paper looking at this problem more generally that you might find interesting https://arxiv.org/abs/2501.19383
I love Mantic's approach to forecasting, but I'm confused about how these Friday Forecast blogs are structured. Are these the direct chain of thoughts of the LLM or simply the final output that reflects on the chain of thought?
Relatedly, how are the quantitative constraints structured? Noting that "I accounted for oil’s inherent volatility by maintaining wide 'tails' in my distribution" still leaves ambiguity as to the extent of the tails as well as the presence and direction of skew.
Hi Isaac! Ben here.
Mantic's forecast output consists of a probabilistic forecast that depends on the question being asked (e.g. a categorical distribution over candidates in an election or, in this case, a probability density over the price of oil) and a summary of its reasoning in prose.
The bullet points in the post are extracted directly from the summary prose. The text in italics is added as background and is not produced by the system. Quantitative descriptions are made concrete in the probability distributions that are produced.
Hey Ben, thanks for the clarification.
I understand if this is a proprietary technique that can’t be divulged, but the method for transforming CoT reasoning / prose into quantitative parameterizations for a log-normal distribution is unclear to me.
In Scott’s talk at ICML, he mentions that the harness prompts the model to output a mixture of log-normal distribution parameters. However, that seems to grant the model enormous numerical freedom, both in choosing the log-normal parameters and the weight given to different scenarios.
For instance, how does the model discern that $mu$ of Scenario A should be $x$ instead of $x + epislon$ or $x + 2 * epsilon$ ? Or, how does the model determine that Scenario B should have weight $w_b$ instead of $w_b + epsilon$ ? (where epsilon is some arbitrary real number)
There are some proprietary techniques at work, but here's a paper looking at this problem more generally that you might find interesting https://arxiv.org/abs/2501.19383