Skip to main content

web access - Parsing JavaScript Script from XML Trees


I've been trying to parse some script written in JavaScript while scraping a website, and am a little confused what functions are needed to get this done in Mathematica.


I imported the data using Import["url","XMLObject"]. The website I am trying to scrape is https://www.msn.com/en-us/money/stockdetails/history/fi-126.1.AAPL.NAS. I am basically trying to scrape historical data prices of certain dates, all of which are hidden in javascript. Any help would be of great appreciation!


The XML for the desired part looks like this:



XMLElement["script", {"type" -> 
   "text/javascript"}, {"\n    require([\"jqLoadTemplate\", \
\"dateFormat\", \"ChartsConfig\", \"chartUtils\", \
\"ChartModeController\", \"utils\", \"LocaleSettings\", \
\"NumberFormatter\", \"c.deferred\"], function ($, dateFormat, \
chartsConfig, chartUtils, ChartModeController, commonUtils, settings, \
numberFormatter) {\n        var configObject = \
chartsConfig[\"StockDetails\"];\n        var chartOptions = \
configObject.chartOptions || {};\n        if (\"False\" === \"True\") \
{\n            configObject.defaultDateRange = \

configObject.defaultOtcRange;\n        }\n\n        function \
loadTable(data) {\n            // Create a table\n            if \
(data != null && data[0] && data[0].seriesId && data[0].series) {\n   \
             var historicalData = [];\n                var size = \
data[0].series.length;\n                var formatString;\n           \
     switch (data[0].Ct) {\n                    case \"1D\":\n        \
            case \"1D_5M\":\n                        formatString = \
settings.shortTimePattern;\n                        break;\n          \
          case \"5D\":\n                        formatString = \
settings.shortTimePattern + \" \" + settings.dayDatePattern;\n        \

                break;\n                    default:\n                \
        formatString = settings.monthDayYearPattern;\n                \
}\n\n                var settingsCopy = $.extend(true, {}, settings);\
\n                settingsCopy.numberDecimalDigits = 0;\n\n           \
     for (var j in data[0].series) {\n                    i = \
data[0].series[size - 1 - j];\n                    var date = \
dateFormat(chartUtils.normalizeDate(i[0]), formatString);\n           \
         var open = numberFormatter.formatNumber(i[4], settings);\n   \
                 var close = numberFormatter.formatNumber(i[1], \
settings);\n                    var low = \

numberFormatter.formatNumber(i[6], settings);\n                    \
var high = numberFormatter.formatNumber(i[5], settings);\n            \
        var volume = numberFormatter.formatNumber(i[3], \
settingsCopy);\n                    historicalData.push({\n           \
             \"date\": date,\n                        \"high\": high,\
\n                        \"low\": low,\n                        \
\"open\": open,\n                        \"close\": close,\n          \
              \"vol\": volume\n                    });\n              \
  }\n                \
$('#containerTable').loadTemplate(('#trTemplate'), historicalData);\n \

           }\n\n        }\n        //dataUrl is URL template. \
urlResolver inserts the parameters.\n        function \
onModeChanged(timeRangeSelected, chartTypeSelected) {\n            \
var dataUrl = configObject.getUrl(timeRangeSelected, \"False\", \
\"False\");\n            var urlResolver = configObject.urlResolver;\n\
            var finalUrl = urlResolver(dataUrl, { symbol: \
\"126.1.AAPL.NAS\", isEOD: \"False\", locale: \"en-US\", \
timeRangeSelected: timeRangeSelected, chartTypeSelected: \"ohlc\", \
isVolumeChartSupported: true });\n            \
chartUtils.getData(finalUrl, configObject.dataBuilder, loadTable, \

function () { }, \"ohlc\");\n        }\n        var timeRange = \
configObject.getTimeRange(\"False\", \"False\", \"126.1.AAPL.NAS\");\n\
            var control = \
ChartModeController.initialize(configObject, onModeChanged, \
\"False\", {}, \"126.1.AAPL.NAS\", timeRange);\n            var \
modeSelector = \
document.getElementById(\"heropanechart-mode-selector\");\n        \
modeSelector.appendChild(control);\n        onModeChanged(timeRange, \
configObject.getDefaultChartType());\n    });\n    "}], "\n ",

Answer




You have misunderstood how this works. What happens is that JavaScript calls a server-side script which returns the data, and the data is subsequently inserted into the HTML. The data cannot be found in the HTML document (in the HTML or in the JavaScript) to begin with.


The most straightforward way to get the data is to monitor the network and observe what URLs the JavaScript code is calling. There is a network tab in Chrome's web developer tool that will do this for you. I opened up the network tab, clicked the "WEEK" tab on the website and took notice of what URL was added to the list of called URLs. It turned out to be this one:


url = "https://finance.services.appex.bing.com/Market.svc/ChartDataV5?symbols=126.1.AAPL.NAS&chartType=5d&isEOD=False&lang=en-US&isCS=true&isVol=true";

The format of the data is JSON. We can import it like this:


data = Import[url, "JSON"];
values = "Series" /. data;

values is now a list of elements such as this:


{"T" -> 10260, "P" -> 141.23, "Hp" -> 141.25, "Lp" -> 141.14, 

 "Op" -> 141.14, "V" -> 34837}

It is unclear what timestamp format is being used, but Hp is definitely the high, Lp the low, Op the open value, p the close value, and V the volume.


To get data for other stocks you have to make the appropriate update to the URL. Changing AAPL.NAS (Apple) to MSFT.NAS (Microsoft) seems to work.


Of course, this is not how the website is meant to be used. They may change the URLs in whatever way they chose, at any moment in time. But this approach may very well work for downloading the data that you want.


Comments

Post a Comment

Popular posts from this blog

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

What is and isn't a valid variable specification for Manipulate?

I have an expression whose terms have arguments (representing subscripts), like this: myExpr = A[0] + V[1,T] I would like to put it inside a Manipulate to see its value as I move around the parameters. (The goal is eventually to plot it wrt one of the variables inside.) However, Mathematica complains when I set V[1,T] as a manipulated variable: Manipulate[Evaluate[myExpr], {A[0], 0, 1}, {V[1, T], 0, 1}] (*Manipulate::vsform: Manipulate argument {V[1,T],0,1} does not have the correct form for a variable specification. >> *) As a workaround, if I get rid of the symbol T inside the argument, it works fine: Manipulate[ Evaluate[myExpr /. T -> 15], {A[0], 0, 1}, {V[1, 15], 0, 1}] Why this behavior? Can anyone point me to the documentation that says what counts as a valid variable? And is there a way to get Manpiulate to accept an expression with a symbolic argument as a variable? Investigations I've done so far: I tried using variableQ from this answer , but it says V[1...
Post a Comment
Read more