postgresql - How can I load incrementally migrate data from PostgresSQL to HDFS? -

-

i have postgresql database use production server. want set hadoop/spark cluster run mapreduce jobs. in order need load data postgres database hdfs. naive approach have batch job once day dumps contents of database (120gb) hdfs. wasteful , costly. since data won't change 1 day next, theoretically cheaper , more efficient send diffs every day. possible?

i've read little sqoop, , seems provide functionality want, requires making changes database , application. there way doesn't require making changes database?

  • apache sqoop can connect postgresql database.

    sqoop provides incremental import mode can used retrieve rows newer previously-imported set of rows, i.e, can table updates happened between previous run , current run.

  • no changes required database.

using sqoop postgresql connector can connect sqoop database , incremental imports without database changes.


Comments

Post a Comment

Popular posts from this blog

c++ - OpenMP unpredictable overhead -

-
i have simple openmp code runs 4 empty threads , reports time: #include "omp.h" #include <sys/time.h> #include <cstdio> double start_time; void process_thread() { } int main(){ struct timeval tv; gettimeofday(&tv, null); start_time = 1000*((double)tv.tv_sec + 1e-6*(double)tv.tv_usec); printf("beginning time: %.3fms ",start_time-start_time); #pragma omp parallel schedule(static,1) for(int i=0;i<4;i++) process_thread(); gettimeofday(&tv, null); double cur_time = 1000*((double)tv.tv_sec + 1e-6*(double)tv.tv_usec); printf("finishing time: %.3fms\n\n",cur_time-start_time); } without openmp running time small , consistent: beginning time: 0.000ms finishing time: 0.050ms beginning time: 0.000ms finishing time: 0.050ms beginning time: 0.000ms finishing time: 0.049ms beginning time: 0.000ms finishing time: 0.049ms beginning time: 0.000ms finishing time: 0.050ms beginnin...
Read more

ruby on rails - RuntimeError: Circular dependency detected while autoloading constant - ActiveAdmin.register Role -

-
i using activeadmin , have file doing this: activeadmin.register role something--- end however, server gives me error: runtimeerror: circular dependency detected while autoloading constant role /library/ruby/gems/2.0.0/gems/activesupport-4.0.10/lib/active_support/dependencies.rb:461:in `load_missing_constant' /library/ruby/gems/2.0.0/gems/activesupport-4.0.10/lib/active_support/dependencies.rb:184:in `const_missing' ~/desktop/boxfox/app/admin/role.rb:1:in `<top (required)>' /library/ruby/gems/2.0.0/gems/activesupport-4.0.10/lib/active_support/dependencies.rb:424:in `load' /library/ruby/gems/2.0.0/gems/activesupport-4.0.10/lib/active_support/dependencies.rb:424:in `block in load_file' /library/ruby/gems/2.0.0/gems/activesupport-4.0.10/lib/active_support/dependencies.rb:616:in `new_constants_in' /library/ruby/gems/2.0.0/gems/activesupport-4.0.10/lib/active_support/dependencies.rb:423:in `load_file' /library/ruby/gems/2.0.0/gems/activesuppor...
Read more

javascript - Wordpress slider, not displayed 100% width -

-
so im making first wordpress theme scratch going fine got problem. want slider on homepage 1 thing wrong. im setting width of image 100% !important doesnt go 100% width. makes image wider not 100%. example here: http://bit.ly/11ouebg see image doesnt contain whole width. can see background-color. really weird, im using meteor slides. i hope can me out :d
Read more