Skip to main content

probability or statistics - Finding a distribution to fit truncated data


I have certain experimental data with the following PDF.


histogram


As you may note the data does not take values lower than 0.


Using FindDistribution I get the following fit:


distribution fit



Is there any good way to fit this kind of "truncated" data with any of the built in distributions?


I tried to use some non negative distributions in Target distributions:


 TargetFunctions -> {LogNormalDistribution, NormalDistribution, 
GammaDistribution, ChiSquareDistribution, HalfNormalDistribution,
ExponentialDistribution, InverseGaussianDistribution}

But didn't get better results.


The data for the Histogram using HistogramList is the following:


{{0., 0.005, 0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.04, 0.045, 0.05,
0.055, 0.06, 0.065, 0.07, 0.075, 0.08, 0.085, 0.09, 0.095, 0.1,

0.105, 0.11, 0.115, 0.12, 0.125, 0.13, 0.135, 0.14, 0.145, 0.15,
0.155, 0.16, 0.165, 0.17, 0.175, 0.18, 0.185, 0.19, 0.195, 0.2,
0.205, 0.21, 0.215, 0.22, 0.225, 0.23, 0.235, 0.24, 0.245, 0.25,
0.255, 0.26, 0.265, 0.27, 0.275, 0.28, 0.285, 0.29, 0.295, 0.3,
0.305, 0.31, 0.315, 0.32, 0.325, 0.33, 0.335, 0.34}, {8.35598,
8.30599, 6.7505, 6.00847, 5.41484, 4.65615, 3.66651, 3.48715,
3.47286, 3.45858, 3.66571, 3.7673, 3.7546, 3.71889, 3.66492,
3.69746, 4.05458, 4.31727, 3.91094, 4.22997, 4.57044, 5.01803,
4.90375, 4.78233, 4.99502, 4.71249, 4.45139, 4.43949, 4.48314,
4.47123, 4.37917, 4.46488, 4.55615, 4.44187, 4.29664, 3.98316,

3.95221, 3.73317, 3.01257, 2.63084, 2.44751, 2.25228, 2.25149,
2.01023, 2.04991, 1.86659, 1.69675, 1.44597, 1.16582, 1.02694,
0.637274, 0.569023, 0.463472, 0.367444, 0.295225, 0.233323,
0.192849, 0.126979, 0.0880914, 0.0253957, 0.0261893, 0.0047617,
0.0198404, 0.0222213, 0.0253957, 0.0388872, 0.0111106, 0.00238085}}

Yo can see the data (around 250.00 points) in the following link:


https://drive.google.com/file/d/0BxzVmMJwPgcNem5xZXFfSnJiQVE/view?usp=sharing



Answer



As you said in the comments, the data for your histograms did contain negatives and you squared all of them. I suggest to consider looking at your values before you square them



Mathematica graphics


Although I have no insight into the underlying process that created the data, one could assume that you have 4 mixed normal distributions here. It seem the left peak is again divide into two distributions. Let us use this for a start and define a mixture distribution:


dist = MixtureDistribution[{a, b, c, 
d}, {NormalDistribution[μ1, σ1],
NormalDistribution[μ2, σ2],
NormalDistribution[μ3, σ3],
NormalDistribution[μ4, σ4]}]

Now we can estimate the parameters by using FindDistributionParameters. I have roughly estimated the initial the initial conditions by just looking where the peaks are, and how high and wide they are:


sol = 

FindDistributionParameters[data,
dist, {{a,
1}, {μ1, -.4}, {σ1, .1}, {b, .3}, {μ2, .1}, {\
σ2, .05}, {c, .5}, {μ3, .4}, {σ3, .1}, {d, .4}, {\
μ4, -.2}, {σ4, .1}}]
(* {a -> 0.376702, b -> 0.125485, c -> 0.275036,
d -> 0.222777, μ1 -> -0.395739, σ1 -> 0.0586256, μ2 ->
0.103749, σ2 -> 0.0496838, μ3 -> 0.337538, σ3 ->
0.0866439, μ4 -> -0.217508, σ4 -> 0.0911891} *)


With 250.000 data values, Mathematica has no problem finding a good fit. Looking at it reveals:


pdf1 = PDF[dist, x] /. sol;

Show[
Histogram[data, {0.03}, "PDF"],
Plot[PDF[dist, x] /. sol, {x, -.6, .6}]
]

Mathematica graphics


Now, since you squared all your values, we need to transform the found mixture distribution as well and you will end up with a distribution that has only values for positive x and fits your data very nicely:



pdf = PDF[TransformedDistribution[u^2, u \[Distributed] dist], x] /. sol;
Show[
Histogram[data^2, {0.007}, "PDF"],
Plot[pdf, {x, 0, .6}]
]

Mathematica graphics


Comments

Popular posts from this blog

plotting - How to draw lines between specified dots on ListPlot?

I would like to create a plot where I have unconnected dots and some connected. So far, I have figured out how to draw the dots. My code is the following: ListPlot[{{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4,13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full] I have thought using ListLinePlot command, but I don't know how to specify to the command to draw only selected lines between the dots. Do have any suggestions/hints on how to do that? Thank you. Answer One possibility would be to use Epilog with Line : ListPlot[ {{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4, 13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full, Epilog -> { Line[ ...

dynamic - How can I make a clickable ArrayPlot that returns input?

I would like to create a dynamic ArrayPlot so that the rectangles, when clicked, provide the input. Can I use ArrayPlot for this? Or is there something else I should have to use? Answer ArrayPlot is much more than just a simple array like Grid : it represents a ranged 2D dataset, and its visualization can be finetuned by options like DataReversed and DataRange . These features make it quite complicated to reproduce the same layout and order with Grid . Here I offer AnnotatedArrayPlot which comes in handy when your dataset is more than just a flat 2D array. The dynamic interface allows highlighting individual cells and possibly interacting with them. AnnotatedArrayPlot works the same way as ArrayPlot and accepts the same options plus Enabled , HighlightCoordinates , HighlightStyle and HighlightElementFunction . data = {{Missing["HasSomeMoreData"], GrayLevel[ 1], {RGBColor[0, 1, 1], RGBColor[0, 0, 1], GrayLevel[1]}, RGBColor[0, 1, 0]}, {GrayLevel[0], GrayLevel...

list manipulation - Selecting multiple columns from a matrix?

Sample data: data = { {{2013, 1, 1}, 24.13, 167.67, 231.82}, {{2013, 1, 2}, 32.15, 170.92, 225.99}, {{2013, 1, 3}, 35.43, 172.68, 221.67}, {{2013, 1, 4}, 36.73, 173.05, 218.32}, {{2013, 1, 5}, 58.19, 165.96, 197.05}, {{2013, 1, 6}, 69.99, 163.50, 187.52}, {{2013, 1, 7}, 71.37, 154.21, 175.58}, {{2013, 1, 8}, 72.51, 149.66, 163.25}}; I want a DateListPlot with three graphs, so for a matrix formed by columns 1 and 2, one for columns 1 and 3, and 1 for columns 1 and 4. At the moment I'm using this code: data2 = Transpose[{data[[All, 1]], data[[All, 2]]}]; data3 = Transpose[{data[[All, 1]], data[[All, 3]]}]; data4 = Transpose[{data[[All, 1]], data[[All, 4]]}]; DateListPlot[{data2, data3, data4}, Joined -> True, Filling -> {3 -> {1}}] but I have a hunch that this can be done more efficiently. I don't like the Transpose s in particular. Any ideas? edit (for extra credit) What if I need to multiply the second column by 2, which in my solution is simp...