Skip to main content

web access - Parsing JavaScript Script from XML Trees


I've been trying to parse some script written in JavaScript while scraping a website, and am a little confused what functions are needed to get this done in Mathematica.


I imported the data using Import["url","XMLObject"]. The website I am trying to scrape is https://www.msn.com/en-us/money/stockdetails/history/fi-126.1.AAPL.NAS. I am basically trying to scrape historical data prices of certain dates, all of which are hidden in javascript. Any help would be of great appreciation!


The XML for the desired part looks like this:



XMLElement["script", {"type" -> 
   "text/javascript"}, {"\n    require([\"jqLoadTemplate\", \
\"dateFormat\", \"ChartsConfig\", \"chartUtils\", \
\"ChartModeController\", \"utils\", \"LocaleSettings\", \
\"NumberFormatter\", \"c.deferred\"], function ($, dateFormat, \
chartsConfig, chartUtils, ChartModeController, commonUtils, settings, \
numberFormatter) {\n        var configObject = \
chartsConfig[\"StockDetails\"];\n        var chartOptions = \
configObject.chartOptions || {};\n        if (\"False\" === \"True\") \
{\n            configObject.defaultDateRange = \

configObject.defaultOtcRange;\n        }\n\n        function \
loadTable(data) {\n            // Create a table\n            if \
(data != null && data[0] && data[0].seriesId && data[0].series) {\n   \
             var historicalData = [];\n                var size = \
data[0].series.length;\n                var formatString;\n           \
     switch (data[0].Ct) {\n                    case \"1D\":\n        \
            case \"1D_5M\":\n                        formatString = \
settings.shortTimePattern;\n                        break;\n          \
          case \"5D\":\n                        formatString = \
settings.shortTimePattern + \" \" + settings.dayDatePattern;\n        \

                break;\n                    default:\n                \
        formatString = settings.monthDayYearPattern;\n                \
}\n\n                var settingsCopy = $.extend(true, {}, settings);\
\n                settingsCopy.numberDecimalDigits = 0;\n\n           \
     for (var j in data[0].series) {\n                    i = \
data[0].series[size - 1 - j];\n                    var date = \
dateFormat(chartUtils.normalizeDate(i[0]), formatString);\n           \
         var open = numberFormatter.formatNumber(i[4], settings);\n   \
                 var close = numberFormatter.formatNumber(i[1], \
settings);\n                    var low = \

numberFormatter.formatNumber(i[6], settings);\n                    \
var high = numberFormatter.formatNumber(i[5], settings);\n            \
        var volume = numberFormatter.formatNumber(i[3], \
settingsCopy);\n                    historicalData.push({\n           \
             \"date\": date,\n                        \"high\": high,\
\n                        \"low\": low,\n                        \
\"open\": open,\n                        \"close\": close,\n          \
              \"vol\": volume\n                    });\n              \
  }\n                \
$('#containerTable').loadTemplate(('#trTemplate'), historicalData);\n \

           }\n\n        }\n        //dataUrl is URL template. \
urlResolver inserts the parameters.\n        function \
onModeChanged(timeRangeSelected, chartTypeSelected) {\n            \
var dataUrl = configObject.getUrl(timeRangeSelected, \"False\", \
\"False\");\n            var urlResolver = configObject.urlResolver;\n\
            var finalUrl = urlResolver(dataUrl, { symbol: \
\"126.1.AAPL.NAS\", isEOD: \"False\", locale: \"en-US\", \
timeRangeSelected: timeRangeSelected, chartTypeSelected: \"ohlc\", \
isVolumeChartSupported: true });\n            \
chartUtils.getData(finalUrl, configObject.dataBuilder, loadTable, \

function () { }, \"ohlc\");\n        }\n        var timeRange = \
configObject.getTimeRange(\"False\", \"False\", \"126.1.AAPL.NAS\");\n\
            var control = \
ChartModeController.initialize(configObject, onModeChanged, \
\"False\", {}, \"126.1.AAPL.NAS\", timeRange);\n            var \
modeSelector = \
document.getElementById(\"heropanechart-mode-selector\");\n        \
modeSelector.appendChild(control);\n        onModeChanged(timeRange, \
configObject.getDefaultChartType());\n    });\n    "}], "\n ",

Answer




You have misunderstood how this works. What happens is that JavaScript calls a server-side script which returns the data, and the data is subsequently inserted into the HTML. The data cannot be found in the HTML document (in the HTML or in the JavaScript) to begin with.


The most straightforward way to get the data is to monitor the network and observe what URLs the JavaScript code is calling. There is a network tab in Chrome's web developer tool that will do this for you. I opened up the network tab, clicked the "WEEK" tab on the website and took notice of what URL was added to the list of called URLs. It turned out to be this one:


url = "https://finance.services.appex.bing.com/Market.svc/ChartDataV5?symbols=126.1.AAPL.NAS&chartType=5d&isEOD=False&lang=en-US&isCS=true&isVol=true";

The format of the data is JSON. We can import it like this:


data = Import[url, "JSON"];
values = "Series" /. data;

values is now a list of elements such as this:


{"T" -> 10260, "P" -> 141.23, "Hp" -> 141.25, "Lp" -> 141.14, 

 "Op" -> 141.14, "V" -> 34837}

It is unclear what timestamp format is being used, but Hp is definitely the high, Lp the low, Op the open value, p the close value, and V the volume.


To get data for other stocks you have to make the appropriate update to the URL. Changing AAPL.NAS (Apple) to MSFT.NAS (Microsoft) seems to work.


Of course, this is not how the website is meant to be used. They may change the URLs in whatever way they chose, at any moment in time. But this approach may very well work for downloading the data that you want.


Comments

Post a Comment

Popular posts from this blog

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more