Algebraic Statistics (IMA2011)

From Macaulay2
Jump to: navigation, search

In the time before the workshop this group should discuss their goals and projects more precisely and narrow to a few key questions. There are general interest overlaps and thus all are listed under this general topic. (This strategy worked well in the Colorado workshop, so it would be a good idea to start the same kind of discussion here and share ideas.)

  • [7Jul2011] For example, one subset of this group is likely to work on the interface between Macaulay 2 and R. Vishesh wrote the interface between R and 4ti2 already, and some of us might want to get acquainted with this as well. --Sonja
  • [20Jul2011] I think that there are some issues with the graphical models package to work on: for example, making sure that the all the functions involving directed/ mixed graphs are compatible (some functions only take mixed graphs as input, but they should be able to handle both mixed graphs and digraphs, or vice versa). Also, I would like to see the addition of undirected graphical models to the graphical models package. --Seth
  • [21Jul2011] FYI: Vishesh will present his R4ti2 interface during the workshop.
  • [23Jul2011] In regards to possible functionality that would be useful from R, it would be nice if a user could run the Metropolis-Hastings algorithm in M2 using a Markov basis. Maybe we can discuss which functions in R are available for this and what we can already do in M2. --Elizabeth

Dynamic Markov Bases

Our goal is to create a package to generate elements of a Markov basis given a specified model on an as needed basis (i.e. instead of generating the entire set of basis elements and choosing one at random). We are starting with decomposable hierarchical models.

July 29th - The basic package is ready and we need to test and document the package.

To do List:

    • Add a lot of documentation and testing
    • Test the package on larger graphs and see how fast it is (a starting point can be n larger than 7)
    • Include MCMC functionality in M2 or have a way to access this package in R
    • Add markov moves for more models for which a closed form solution is known

Redoing the Gaussian Graphical Models part of GraphicalModels.m2 Package

  • Separate the computation of graphical models vanishing ideal from the computation of ideals defined by t-separation. Give these functions new names.
  • Can't currently compute both the vanishing ideal of a directed graph and the t-separation ideal in the same Gaussian ring.
  • Make all Gaussian graphical model computations work for all graph classes (mixed graph, bigraph, digraph, and graph)
  • Need to add material for doing computations when graphs have cyclic part (in the directed part) or have undirected edges.
  • TO DO: 26JULY2011 make all gaussianRing methods take specific variablenames as optional inputs, NOT as one long list.
  • TO DO: add error messages everywhere where necessary when the ring is not a gaussianRing . this is the line to be added

if not R#?gaussianRing then error "expected a ring created with gaussianRing"; to the following methods:

    • marginMap needed markovRing
    • hiddenMap needed markovRing
    • markovMatrices needed markovRing
    • covarianceMatrix for mixedGraph
    • directedEdgesMatrix
    • bidirectedEdgesMatrix
    • gaussianParametrization
    • gaussianRing
    • covarianceMatrix
    • undirectedEdgesMatrix
    • other undirected gaussian methods -they just end up calling the two methods right above so no need to double-check.
  • TO DO: for every method where a graph/digraph and a ring are inputs, check that the variable names match so that the method can actually be executed.(think: the ring R may not always be obtained by R=gaussianRing G; it could also be a bigger graph's ring.) Methods needing this change:
    • conditionalIndependenceIdeal(Ring,Graph)
    • any method that uses graph as input?
  • TO DO: make the declaration of all Gaussian rings include the graph. The undirected ring now carries with it the graph: R.graph. Also, the directed ring now carries with it the graph: R.graph. SEE SEP 8 NOTES number 4.
  • TO DO; Change all methodss that take as input a Gaussian ring and a graph, just the input of the Gaussian ring should be needed. (Currently, an input that requires both R and G in most functions is just using G to recover the names of the vertices.) we will use the functionality of the bullet above
  • TO DO: similar to above, but for digraphs. In particular, there are problems when the digraph G is simply ignored by the method if it is passed along the ring...-- see for example the comments in the code in covarianceMatrix(Ring,Digraph). ---I believe most of this has been fixed now -if not, code contains comments...

Status report (Thursday):

  • Elizabeth and Vishesh working on DynamicMarkovBases.m2 for hierarchical models. Completed the following methods:
    • isChordal--checks if a graph is chordal
    • peo--returns a perfect elimination ordering given a graph
    • baseSets--returns base sets and their corresponding dependent vertices
    • separators--returns a list of one or two element minimal vertex separators
    • markovMove-- returns a uniformly random markov move for a decomposable graphical model
  • Seth and Sonja added methods to GraphicalModels.m2 to take care of undirected Gaussian case:
    • gaussianRing,
    • covarianceMatrix,
    • localMarkov
    • pairMarkov
    • globalMarkov
    • undirectedEdgesMatrix
    • conditionalIndependenceIdeal
    • gaussianVanishingIdeal
  • David adding tests and documentation nodes for the new undirected functions:
    • gaussianRing
    • undirectedEdgesMatrix
    • covarianceMatrix
    • gaussianVanishingIdeal
    • pairMarkov
    • localMarkov
  • Sonja added documentation nodes for
    • conditionalIndependenceIdeal
    • sVariableName and 3 other similar crappy names.

Status report (Friday):

  • GraphicalModels.m2:
    • changed Markov matrices to take as input a list of vertex labels instead of Graph.
    • updated c.i. ideal to reflect this
    • deleted markovIdeal.
    • Fixed gaussianRing to look like markovRing in the sense that htere is now a hidden hash table called gaussianRingList that stores all previously created gaussianRings, indexed by the coefficient rings, variable names, and number of nodes (or vertices of digraph or graph).
    • Fixed covarianceMatrix to be streamlined (only accepts ring as input. gaussian ring.) EXCEPT when input is mixedGraph - we did not touch this!
    • TO DO: document these changes:
      • markovMatrices ,
      • conditionalIdndependenceIdeal,
      • covarianceMatrix .
    • TO DO: nothing has been done for mixedGraphs in terms of streamlining, fixing silly input, etc etc.


Active to-do list (created 8 Sep 2011. updated 29Nov2011. Update July 2012 !! ):

  • 1. Delete gaussianIdeal method and its documentation node. Make sure the method is not called from any other method in the package. (If it is, note that it is replaced by conditionalIndependenceIdeal, so change the package as necessary.) (also deleted gaussianMinors) -15sep2011
  • 2. Finish documenting conditionalIndependenceIdeal (see line number 2162 in the package!). In particular, show how to use the method with (Ring, List, List) to explain why and how you'd wanna input VarNames: it is only there for the discrete case. The point is that there are no variable names in markovRing. (this is a "to-do" item in the documentation now.)
  • 3. Sonja has changed line 601 to say R.digraph =G (it used to say R.graph = G). Check to see if this broke other parts of the package.
    • (idea: the graph/digraph is carried around with the ring so the user doesn't have to input the graph/digraph over and over... for the purpose of the IMA workshop, there was no need to distinguish between R.graph and R.digraph case. But now suddenly there is a need. Test all methods that use R.graph for directed case).
    • update conditionalIndependenceIdeal(R,Digraph) and conditionalIndependenceIdeal(Ring,List)
    • document CI ideal for digraph
  • 4. consolidate covarianceMatrix and covarianceMatrixUndirected into 1 method
  • 5. delete method markovIdeal
  • 6. Make conditionalIndependenceIdeal for discrete case!!!
  • 7. I do not understand the use of VarNames in conditionalIndependenceIdeal, after all :( just looking at the examples from the documentation, I'm not sure what VarNames does. Answer: b/c you want to be able to give CI statements for arbitrary random variable names!!
  • 8. for every method where a graph/digraph and a ring are inputs, check that the variable names match so that the method can actually be executed.(think: the ring R may not always be obtained by R=gaussianRing G; it could also be a bigger graph's ring.) Methods needing this change:
    • conditionalIndependenceIdeal(Ring,Graph)
  • 9. DONE! Make sure that we do not force the user to input the graph or digraph if it's already been stored in gaussian ring. -----DONE for all relevant instances of (Ring, Graph). For input of the form (Ring, Digraph), see the following:
  • 9(a). fix gaussianMatrices - the old method is crap and should be deleted; use the code from conditionalIndependenceIdeal to do this in a smarter way.
  • 9(b). trekIdeal Note: we also moved the Digraph part of this to gaussianVanishingIdeal.
  • 10. I wonder if we still need conditionalIndependenceIdeal(Ring,Graph) and (Ring, Digraph), since there are so many "if-else's" in the method itself - overload? CHECK THIS.
  • 11.DONE!! When Stmts are used, check if variable names match those in the ring and print error message "variables names in statements do not match list of random variable names". WE MAY NEED TO make a change in the following methods (check each!):
    • gaussianMatrices
    • conditionalIndependenceIdeal(Ring,List) -- what if the vars in ring do not match the vars in Stmts??
    • trekIdeal
  • 12. obsolete!
  • 13. gaussianRing of mixedGraph needs to be smarter (hashtables!); insert R.mixedgraph as placeholder
  • 14. trekIdeal works for mixedGraph; needs to work for Graph and Digraph This works now, but still need to add one last error check in the case that the gaussianRing was declared as gaussianRing n. Need to check vertex labels of the graph match this. --Seth
  • 15. bug from Luis/Mike Benfield: G = digraph { {1, {2}} }; R=gaussianRing G; conditionalIndependenceIdeal(R,G) --returns integer instead of the zero ideal. Luis has fixed this bug on 10/25/11. It still needs to be checked against a graph that generates a non-empty list of independent statements none of them generating nonzero minors. I think my fix will handle this case, but it needs to be tested. --Luis
  • 16. DONE!! While working on trekIdeal, discovered that many functions related to mixed graphs require input of ring and graph, when should require only input of ring (e.g. covarianceMatrix, directedEdgesMatrix, bidirectedEdgesMatrix). I have fixed this for covarianceMatrix, but it is still needed for other functions. --Seth. -- Changed the three methods so they don't require MixedGraph. Fixed the rest of the code to reflect changes. --Sonja --Updated documentation to reflect changes. --Luis
  • 17. Another thing I noticed: conditionalIndependenceIdeal doesn't work at the moment when you have a gaussianRing declared with a mixed graph Luis fixed this.
  • 18. DONE! Currently conditionalIndependenceIdeal(R,G) where G is a graph or digraph doesn't work properly. We should think about deleting those functionalities, since you could just do conditionalIndependenceIdeal(R, globalMarkov(G)) instead. --agree with this; documentation should be updated to reflect this and just tell people what to do. -- Might need to add some checks to conditionalIndependenceIdeal to check things? --Seth (see point number 11 !!! --Sonja)
  • 19. About 7 FOUR TESTS FAIL DUE TO ORDERING; some because of unordered output of localMarkov for example. Need to find a clever way to run tests (check containment of statements quickly up to reordering). -- we will fix this by making semi-test; i.e. instead of testing whether the entire list of statements is correct up to reordering (!), we will just check whether each statement makes sense (whether appropriate minors vanish).
  • 20. Perhaps we should "unhide" the following fields for a gaussianRing R: R.graph, R.digraph, R.mixedgraph. Other then for programming, being able to access these might be useful? (Sonja)
  • 21. Decide if we want to document or to hide (make local) the field numberOfEliminationVariables !
  • 22. Seth's example :)
  • 23. -- This package will be finalized in August 2012. Stay tuned! --

Return to the Main IMA2011 page


August 2012 - GraphicalModels.m2

Seth, Luis, Sonja meeting:

  • 1. DONE: gaussianVanishingIdeal does not yet work for MixedGraph
  • 2. DONE: return error whenever there is a directed cycle (use isCyclic from Graphs). The following methods need this:
    • pairMarkov
    • localMarkov
  • 3. DONE; Implement Seth's recursive computation...
  • 3a. DONE: test this recursive function with changed variable names does indeed work.
  • 3b. DONE: add documentation node + test for this thing.
  • 4. DONE: Fix the old test failing problem: four tests fail due to ordering; some because of unordered output of localMarkov for example. Need to find a clever way to run tests (check containment of statements quickly up to reordering).
  • 5. DONE: Update entire documentation. (including main documentation node.) All of the nodes need updating! Left:
    • identifyParameters
    • trekIdeal
    • trekSeparation
    • undirectedEdgesMatrix
    • gaussianVanishingIdeal
    • problem - posted to google groupVariableName --> move (if possible) to standalone after markovRing can't be done at the moment. need help.
    • problem - posted to google groupsVariableName etc. --> moved (if possible) to standalone after gaussianRing can't be done at the moment. need help.
  • 6. OK - we don't care: identifyParameters only works with mixedGraph. OK? - ok.
  • 7. DONE: local markov and gaussianRing changed to local markovRingData and gaussianRingData. (!) Updated all instances in the code. Made gaussianRingData also be accessible with a "." instead of "#".
  • 8. DONE: Check if more tests are needed.
  • 9. DONE: local/global functions like "cartesian". see note in code.
  • 10. DONE: removed the graph/digraph/mixedgraph as required input from the following methods (and updated doc, tests):
    • gaussianParametrization
    • identifyParameters
  • ...
  • n. Most general goal:
    • One day: Make all Gaussian graphical model computations work for all graph classes (mixed graph, bigraph, digraph, and graph).
    • One day: Need to add material for doing computations when graphs have cyclic part (in the directed part) or have undirected edges.