Skip to main content

probability or statistics - Finding a distribution to fit truncated data


I have certain experimental data with the following PDF.


histogram


As you may note the data does not take values lower than 0.


Using FindDistribution I get the following fit:


distribution fit



Is there any good way to fit this kind of "truncated" data with any of the built in distributions?


I tried to use some non negative distributions in Target distributions:


 TargetFunctions -> {LogNormalDistribution, NormalDistribution, 
  GammaDistribution, ChiSquareDistribution, HalfNormalDistribution, 
  ExponentialDistribution, InverseGaussianDistribution}

But didn't get better results.


The data for the Histogram using HistogramList is the following:


{{0., 0.005, 0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.04, 0.045, 0.05,
   0.055, 0.06, 0.065, 0.07, 0.075, 0.08, 0.085, 0.09, 0.095, 0.1, 

  0.105, 0.11, 0.115, 0.12, 0.125, 0.13, 0.135, 0.14, 0.145, 0.15, 
  0.155, 0.16, 0.165, 0.17, 0.175, 0.18, 0.185, 0.19, 0.195, 0.2, 
  0.205, 0.21, 0.215, 0.22, 0.225, 0.23, 0.235, 0.24, 0.245, 0.25, 
  0.255, 0.26, 0.265, 0.27, 0.275, 0.28, 0.285, 0.29, 0.295, 0.3, 
  0.305, 0.31, 0.315, 0.32, 0.325, 0.33, 0.335, 0.34}, {8.35598, 
  8.30599, 6.7505, 6.00847, 5.41484, 4.65615, 3.66651, 3.48715, 
  3.47286, 3.45858, 3.66571, 3.7673, 3.7546, 3.71889, 3.66492, 
  3.69746, 4.05458, 4.31727, 3.91094, 4.22997, 4.57044, 5.01803, 
  4.90375, 4.78233, 4.99502, 4.71249, 4.45139, 4.43949, 4.48314, 
  4.47123, 4.37917, 4.46488, 4.55615, 4.44187, 4.29664, 3.98316, 

  3.95221, 3.73317, 3.01257, 2.63084, 2.44751, 2.25228, 2.25149, 
  2.01023, 2.04991, 1.86659, 1.69675, 1.44597, 1.16582, 1.02694, 
  0.637274, 0.569023, 0.463472, 0.367444, 0.295225, 0.233323, 
  0.192849, 0.126979, 0.0880914, 0.0253957, 0.0261893, 0.0047617, 
  0.0198404, 0.0222213, 0.0253957, 0.0388872, 0.0111106, 0.00238085}}

Yo can see the data (around 250.00 points) in the following link:


https://drive.google.com/file/d/0BxzVmMJwPgcNem5xZXFfSnJiQVE/view?usp=sharing



Answer



As you said in the comments, the data for your histograms did contain negatives and you squared all of them. I suggest to consider looking at your values before you square them



Mathematica graphics


Although I have no insight into the underlying process that created the data, one could assume that you have 4 mixed normal distributions here. It seem the left peak is again divide into two distributions. Let us use this for a start and define a mixture distribution:


dist = MixtureDistribution[{a, b, c, 
   d}, {NormalDistribution[μ1, σ1], 
   NormalDistribution[μ2, σ2], 
   NormalDistribution[μ3, σ3], 
   NormalDistribution[μ4, σ4]}]

Now we can estimate the parameters by using FindDistributionParameters. I have roughly estimated the initial the initial conditions by just looking where the peaks are, and how high and wide they are:


sol = 

 FindDistributionParameters[data, 
  dist, {{a, 
    1}, {μ1, -.4}, {σ1, .1}, {b, .3}, {μ2, .1}, {\
σ2, .05}, {c, .5}, {μ3, .4}, {σ3, .1}, {d, .4}, {\
μ4, -.2}, {σ4, .1}}]
(* {a -> 0.376702, b -> 0.125485, c -> 0.275036, 
 d -> 0.222777, μ1 -> -0.395739, σ1 -> 0.0586256, μ2 ->
   0.103749, σ2 -> 0.0496838, μ3 -> 0.337538, σ3 -> 
  0.0866439, μ4 -> -0.217508, σ4 -> 0.0911891} *)


With 250.000 data values, Mathematica has no problem finding a good fit. Looking at it reveals:


pdf1 = PDF[dist, x] /. sol;

Show[
 Histogram[data, {0.03}, "PDF"],
 Plot[PDF[dist, x] /. sol, {x, -.6, .6}]
 ]

Mathematica graphics


Now, since you squared all your values, we need to transform the found mixture distribution as well and you will end up with a distribution that has only values for positive x and fits your data very nicely:



pdf = PDF[TransformedDistribution[u^2, u \[Distributed] dist], x] /. sol;
Show[
 Histogram[data^2, {0.007}, "PDF"],
 Plot[pdf, {x, 0, .6}]
 ]

Mathematica graphics


Comments

Post a Comment

Popular posts from this blog

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more